Time-Reversal Phase Transition at the Edge of 3d Topological Band Insulator 
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We study the time reversal (T) symmetry breaking of 2d helical fermi liquid, with application 
to the edge states of 3d topological band insulators with only one two-component Dirac fermion at 
finite chemical potential, as well as other systems with spin-orbit coupling. The T-breaking Ising 
order parameter is not over-damped and the theory is different from the ordinary Hertz-Millis theory 
for order parameters at zero momentum. We argue that the T-breaking phase transition is an 3d 
Ising transition, and the quasiparticles are well-defined in the quantum critical regime. 

PACS numbers: 



Time reversal (T) symmetry is the key to guarantee 
the stability of both 2d and 3d topological insulators 
(TBI) [2-13], therefore it is meaningful to study the T- 
symmetry breaking in these systems. Because the bulk 
of TBI is always an insulator, the T-breaking transition 
only involves the edge states, which are gapless in a T- 
symmctric phase, and the spectrum opens up a gap when 
T is broken. The simplest version of 3d TBI has only 
one two-component Dirac fermion at the edge, which 
can be perfectly realized in materials based on Bi2Se3 
and Bi 2 Te3 |5j-[8j]. The time reversal symmetry can ei- 
ther be broken explicitly by magnetic impurities, or bro- 
ken spontaneously by strong enough interactions. The 
effects of magnetic impurities and quenched disorders on 
the edge states of 2d and 3d TBI has been discussed in 
Ref. [1, H3] and Rcf. 11 1 respectively. Spontaneous 



T-breaking phase transition is most relevant to the tran- 
sition metal version of the 3d TBI with interplay between 
spin-orbit coupling and strong interaction [12[, and it is 
the goal of the current paper. 

Without loss of generality, the edge state of 3d TBI 
is described by the following time-reversal invariant La- 
grangian dEl: 



L f = ip("fo(idt - M) + Vfijjdj)^. 



(1) 



7° = a z , 7 1 = icr x , 7 2 = ia v , ?/> = V^7°- v f is the 
fermi velocity at the Dirac point, /i is the chemical po- 
tential. The Pauli matrices in Eq. [I] represent the pseu- 
dospin, which is a combination between real spin space 
and orbital space. For conciseness we will call a a the 
spin hereafter. The spin a a of the electrons are per- 
pendicular with their momenta. This helical spin align- 
ment has been successfully observed in a recent ARPES 
measurement The T-symmetry guarantees that in 

the Lagrangian the Dirac mass gap -tpip does not appear 
explicitly, although a mass generation can occur when 
the T-symmetry is spontaneously broken. The Dirac 
gap is simply the z— spin magnetization, hence the gap 
can be spontaneously generated with strong enough fer- 
romagnetic interaction between z— component of spins: 
— ('iljip) r 'VF y F>(if}ip)j*. To describe this T-breaking transi- 
tion, we can define an Ising order parameter <f>, which 



couples to the Dirac fermions as 
L = Lf + Lb + Lbf, 



L b = {d t ^) 2 - J2 rf^) 2 - rcf? ~ u<j> 4 

i—x,y 

L bf = gW>i>. 



(2) 



ijjijj order breaks T, and drives the edge to a quantum 
Hall phase. Identifying the leading spin order instability 
requires detailed knowledge of the fermion interaction, 
hence we focus on the universal physics at the quantum 
critical point, assuming the existence of the phase tran- 
sition. In the current work we only discuss the discrete 
symmetry breaking, the transition with continuous sym- 
metry breaking will be studied in another paper [151 ] . The 
Lagrangian Eq. [5] can also describe the phase transition 
of magnetic impurities doped into the system, and the 
order parameter (f> stands for the global magnetization 
of the magnetic impurities. The w</> 4 term represents ei- 
ther the self-interaction between the magnetic impurities, 
or the higher order spin-spin interactions between heli- 
cal fermions. In this paper we assume u > and large 
enough to ensure a second order transition. 

Let us first take fi = in Eq. [21 now this model 
becomes the Higgs- Yukawa model, which is believed to 



is im l 



be equivalent to the Gross-Neveu model 
i^iid^ip — 7( - 0V') 2 ; & t least when Vf — Vb- The tran- 
sition of <j) is not 3d Ising transition because the cou- 
pling g is relevant at the 3d Ising fixed point, based 
on the well-known scaling dimensions [ip] = 1/2, and 
\4>] = (d - l)/2 + 7?/2 = 0.518 at the 3d Ising fixed point 
|18| . If there are N flavors of Dirac fermions, The critical 
exponents of this transition with large N have been calcu- 
lated by means of 1/N and e = 4 — d expansions [l9|-[22j], 
and a second order transition with non-Ising universality 
class was found. In our current case with N = 1, there 
is no obvious small parameter to expand, we conjecture 
that the transition is still second order, with different 
universality class from the 3d Ising transition. 

Let us now turn on a finite chemical potential /1, but 
still make /i much smaller than the band- width 2A of the 
edge states. Now the edge states become a helical fermi 



a b 

FIG. 1: The fermi surface of Dirac fermions, with finite chem- 
ical potential, a, when we translate the fermi surface with a 
small momentum q, at the intersection the spins are almost 
parallel; b, the two patches of fermi surface Eq. HJ] describes. 

liquid, with spins aligned parallel with its fermi surface. 
The tuning parameter r in Eq. [2] will be renormalized by 
the static and uniform susceptibility of a z of the helical 
fermi liquid 

Ar0 2 =Re[x(O,O)]0 2 ~ 5 2 (M-A)<£ 2 . (3) 

Therefore the phase transition of 4> can be driven by 
tuning the chemical potential /i. Also, it is straightfor- 
ward though a little tedious to check that the momen- 
tum and frequency dependence of Rc[x] are nonsingular: 

Re[\(uJ, q)} ~ c - c x u? - c 2 q 2 H . 

As the ordinary Hert-Millis theory [23| of quantum 
phase transition inside fermi liquid, the singular correc- 
tion to the effective Lagrangian of order parameter comes 
from the imaginary part of the susceptibility. At the crit- 
ical point, the critical mode of Ising order parameter (f> 
can be damped through particle-hole excitations. The 
damping rate can be calculated from the Feynman dia- 
gram Fig. [2ji, or through the Fermi- Golden rule 

Tmpifaq)] ~ J -fjL[f( ek+q ) - f(e k )} 

x 5(\u\ - e k+q + e k )\(k\gip k ^ k+q \k + q)\ 2 

~ 9 2 ^J^4- 2 - (4) 
Vfkj]] vjq 2 

This result is obtained in the limit q <C fe/, kf is the fermi 
wave-vector. When \oj\ > Vfq the scattering rate vanishes 
for kinematic reasons, therefore when vt, > Vf this decay 
rate is unimportant because the Green's function of <f> 
will peak when uj ~ v^q. From now on we will assume 
that Vb < Vf. The decay rate obtained above differs from 
the Hertz-Millis theory [23| which usually takes the form 
\uj\/q for order parameters at zero momentum. This re- 
sult can be physically understood as following: <j)(q) can 
transfer momentum q to the fermi surface, and if we de- 
note the fermi surface as So, and denote the fermi surface 
translated by a small momentum q as S^, then as long as 
q is small enough and Sq will have almost the same 
spin directions at their intersection. Because a z always 
flips spin direction in the XY plane, when two spins are 




FIG. 2: The one-loop Feynman diagrams for boson, fermion 
self-energy, vertex correction, and (j> n term generated with 
fermion loop. The dashed line and solid line represent the <j> 
propagator and fermion propagator respectively. 

parallel the matrix element of a z vanishes. Mathemati- 
cally this intuition is manifested as \(k\i[) k %l) k+q \k + q)\ 2 
vanishes as q 2 in the limit of q — > 0. Therefore in this case 
<j) is not overdamped at low momentum and frequency. 

If we ignore the self-interaction between (f>, and take 
the Gaussian part of we can calculate the self-energy 
correction of fermion ip through Feynman diagram Fig. 

Evaluated close to \v\ ~ e 9 , the imaginary part of 
fermion self-energy scales as 

Yj(v)" — / d 2 k— [6(e k+q )5(v - e k+q - co k ) 
J w fe 

- 9(-e k+q )S(v - e k+q + uj k )} 

x \(q\g$ q il> k +q\k + q)\ 2 ~<?VSignM + ■ • -(5) 

Unlike the Hertz-Millis theory, the scaling of is 
similar to fermi liquid, which means that the quasiparti- 
cles are well-defined even at the quantum critical point. 

The above calculations are only one-loop level. To 
evaluate higher loop diagrams, we had better simplify 
the problem by considering two patches of the fermi sur- 
face around two opposite points (±/c/,0), and label the 
fermions in terms of its momentum p x — k x ~kf, p y = k y . 
Now the action becomes 

L f = V f (p)(w - v f p x T z - v y pl)ip(p), 

L b = V u 2 \t(p)\ 2 - (v 2 bx pi + v^pDimi 2 +■■■ 

L bf = igq v H<t)^(p)T z ip{p + q) + --- (6) 

Here both \p x \ and \p y \ are much smaller than kf, and 
v y = Vf/(2kf). t z is the Pauli matrix operating on the 



3 



space of two fermi patches (±fc/,0). This isolated patch 
approximation is based on the observation that <pg most 
strongly couples to the patch with q ± Kf, where the 
particle-hole excitation with momentum q is soft. Also, 
at low energy limit, none of the scattering process will 
mix these fermions with those from other patches. For 
instance, if we integrate out the boson <p$, interaction be- 
tween different patches will be induced, but the standard 
scaling argument for ordinary fermi liquid suggests that 
the only important interaction at low energy has q = 
i.e. the 5ngdng> interaction. However, when q = the 
interaction vertex vanishes. Therefore the isolated patch 
approximation is reasonable. 

Under discrete symmetry transformations T, P x and 
P y , the physical quantities in Eq. [B] transform as 



T 

P:, 
P„ 



t — > —t, ip t x %\), ki — ► — hi, 4> -> - 
x — > —x, ip — > r x ip, k x — > —k x , 4> — > 
y -)■ —y, ij) -tip, k y -4 -k y , <p -> —4 



(7) 



and the action is invariant. Had we only kept one single 
fermi patch at (+&/, 0) like Ref. [24|,l25| , the action would 
not be invariant under these discrete transformations. 

The fermion-boson vertex is proportional to q y of </>, 
therefore for any loop diagram with <p external line, the 
loop diagram will vanish as q y — > for each <p exter- 
nal line. There are two different ways to assign scaling 
dimensions to operators in Eq. [6] 

Scaling 1, [u] = 1, [p x ] = 1, [p y ] = 1, [v y ] = -1, 

[4>] = -~, bP] = -2, W = [v bx ] = [v by ] = 0, 



Ifj] 



2 
1 

2' 



Scaling 2, [u] = 2, \p x ] = 2, \p y ] = 1, [v y ] = 0, 



m = m = -\ h = K] = -2, 

\9] = -\. 



"by- 



= 0, 



(8) 



For both scaling choices, [g] < 0, i.e. according to the 
naive scaling the coupling between fermions and bosons 
are irrelevant, and the loop diagrams are suppressed. 
When we evaluate loop integrals, irrelevant terms can 
in general be ignored, but in order to avoid divergence 
from integrating a constant, we have to make a diagram- 
dependent choice of scaling from the two options in Eq. 
[HI otherwise some irrelevant terms have to be kept in 
the integral. For instance, we can reproduce the results 
obtained previously from scaling argument: at the g 2 or- 
der, choosing the second scaling in Eq. |8l the self-energy 
correction of <fi should have dimension 3, which is consis- 
tent with the direct calculation with action Eq. [5] and 
Feynman diagram Fig. |2Jx: 



Im[Xy ~ g 2 \u\\q y \, 



(9) 



which due to energy conservation is valid when \u> — 
Vfq x \ < Vf\q y \. For the fermion self-energy, in order to 



avoid naive divergence one has to choose the first set of 
scaling dimensions, [g 2 ] = — 1 implies that the self-energy 
should have dimension 2, which is consistent with the re- 
sult S(V)" ~ g 2 v 2 we obtained before. The one loop ver- 
tex correction can be calculated using the second scaling 
and Fig. [5]/, the result is V q — ql/{\q y \ + cg 2 \u>\). 

Now let us discuss the nature of the T-breaking tran- 
sition. The pure boson Lagrangian Lb in Eq. [2] describes 
a 3d Ising transition. At the g 2 order the perturbation 
at the 3d Ising transition is included in the self-energy 
correction to <p, whose singular contribution is in the 
imaginary part. The imaginary part of the self-energy 
is given by both Eq. [4] and Eq. HI evaluated with the 
the full fermi surface and isolated patch approximation 
respectively. In both cases this self-energy mix (j> at dis- 
tinct points in space-time, their actual scaling dimen- 
sions at the 3D Ising critical point can be estimated as 
D - (2 + D - 2 + 77) = -ri, j] - 0.037 [TJ]. Therefore at 
the g 2 order there is no relevant perturbation induced at 
the 3d Ising fixed point. 

The higher loop diagrams are more complicated, al- 
though in the previous paragraph we showed that in both 
choices of scalings g is irrelevant, it does not immedi- 
ately imply none of the higher order loops can generate 
important terms at the 3d Ising fixed point. This is be- 
cause when we evaluate the fermi loop, in order to avoid 
naive divergence we have to take the second scaling in Eq. 
[8l which is different from the 3d Ising fixed point with 
isotropic scaling dimensions in space-time. For instance 
the leading <p n term generated at g n order perturbation 
is given by diagram Fig. |2fc, which should take the form 



(10) 



Notice that all the <p n terms with n odd are forbidden by 
symmetry. This term is irrelevant based on the second 
scaling of Eq. [51 but in order to know its scaling dimen- 
sion at the 3d Ising fixed point, we need to evaluate its 
form more explicitly. The function f(uij, qj) is integral of 
the following fermion loop: 

fn(uj,qj) ~ J dudp x dpy x S(y]u)j)5(y]qj) 

n j j 

j — l i—l i—1 

After the integral, this term has a very complicated de- 
pendence of the external frequency ujj and momentum pj , 
but since we are only interested in its scaling dimension, 
the following schematic form will be good enough: 



101 



E\Qv\I%=t(toj+vfQj*) 



(12) 



and Q represent linear combination between external 
frequency and momentum q respectively. In the denomi- 
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nator, the ellipses include terms with higher power of mo- 
mentum compared with the leading term. We can easily 
verify that when n = 2 Eq. [T^] reproduces the well- 
known result |oj l/loVr I - At the 3d Gaussian fixed point, 
the coefficient of the </>™ term will have scaling dimension 
1 — n/2, which should be irrelevant for any n > 4. Eq. Q~2] 
is applicable to the kinematic regime with all the exter- 
nal momenta nearly parallel to y, when <f> couples most 
strongly with particle-hole excitations. For more general 
kinematic regime the <f) n term generated is expected to 
be no more singular than Eq. 1121 

So far we have only considered the leading <j) n term, 
which is generated at g n order. Higher order contribution 
to 4> n always involve one or more internal boson lines 
like Fig. and because of the suppression of p y at the 
internal vertices, we expect these higher order terms will 
not be more relevant than the leading order. For instance 
the result of diagram Fig. ^jp with one internal boson 
line has the same scaling dimension as Fig. [2h. Based 
on these observations, the T-breaking phase transition in 
the helical fcrmi liquid with finite /i is expected to be a 3d 
Ising transition. If we take into account of the interaction 
between <f> at the 3d Ising universality class i.e. using 
the fully dressed boson propagator in Fig. [2]i, the self- 
energy of the fermion will be even more suppressed due 
to self-screening between bosons. One reasonable result 
could be E(z/)" ~ g 2 \y\ 2+r iSiga[v], r\ ~ 0.037 [18] is the 
anomalous dimension of <f> at the 3d Ising transition, since 
rj > 0, the quasiparticle is always well-defined at the 
quantum critical regime. 

The Lagrangian Eq. [I] is invariant when spin and space 
are rotated by the same and arbitrary angle, which is 
generically larger than the symmetry of the microscopic 
system. For instance in material E^-zSn^Tea the fermi 
surface of edge states is not circular when the chemi- 
cal potential is large, instead it is a hexagonal star with 
six sharp corners [8|. Therefore with large chemical po- 
tential, terms with higher order momentum should be 
considered in the free electron Lagrangian Lf of Eq. Q] 
These higher order terms can lead to many new effects, 
for instance it may align the spins slightly along z direc- 
tion instead of completely within the XY plane (26. 27|. 
although the integral of a z vanishes along the whole fermi 
surface. If the spins have z component, then <p ~ i/jtp will 
cause a deformation of the fermi surface, and is over- 
damped for small momentum, in this case the ordinary 
z = 3 Hertz-Millis theory becomes applicable. 

In summary, we studied the time-reversal symmetry 
breaking for single Dirac fermion with finite chemical po- 
tential. Unlike the ordinary Hertz-Millis theory, the Ising 
order parameter is not overdamped, and we argue that 
the coupling between Ising order parameter and fermions 
is weak in the infrared limit. The transition most likely 
belongs to the 3d Ising universality class. The analysis 
in our paper can be generalized to many other systems. 
For instance we can consider the spin order ■ip' f a z ip in 



the Rashba model [28|, l29( with inner and outer fermi 
surfaces with opposite inplane helical spin direction, and 
the results are very similar to our paper. Another sys- 
tem is graphene with N — 4 flavors of Dirac fermion, 
our analysis applies to order parameters tp^ and TpT a ijj 
(T a 6 SU(iV)). For instance the phase transition of 
Quantum Spin Hall order ^Sip belongs to the 3d 0(3) 
universality class, when the fermi energy is tuned away 
from the Dirac point. In future we shall try to make 
connection between our results and realistic physical sys- 
tem, after a suitable physical system with both topolog- 
ical band structure and strong interaction is discovered, 
like the one studied theoretically in Ref. 12j. 
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